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(54) Interactive data processing apparatus using a pen. 

@ Information is inputted to a data processing apparatus by using a pen (3) in conjunction with an input 
device or tablet (2) which is mounted on a display device or panel (1). If the user touches an icon (e.g. 
41), selected from one or more groups (40) of icons displayed on the display device (1) and indicating 
various functions, with the point of the pen (3), drags the pen (3) along the surface of the display device 
(1) to a display position, and then releases the point of the pen from the surface of the display device, an 
enlarged processing display form of the icon (41), in the form of a window (43), is automatically 
displayed at a desired position on the display device (1). Further, if the user touches the window (43) 
with the point of the pen (3) and drags the pen along the surface of the display portion to the position of 
a desired icon, the enlarged window is automatically reduced and is stored in a vacant icon. 
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The invention relates to data processing 
apparatus and, more particularly, is directed to con- 
trolling the inputting of information to data processing 
apparatus. 

Conventionally, a keyboard, a mouse and/or the 
like are generally used as an information input control 
apparatus of a computer. For example, if the user 
wants to enlarge a window of an icon on a picture 
screen of a display portion such as a cathode ray tube 
(CRT) or the like by using a mouse, the mouse can be 
made operable on a tablet and a cross-shaped posi- 
tion designation cursor is moved on the picture screen 
of the display portion by moving the mouse. When a 
button on the mouse or a predetermined key on a 
keyboard is depressed once in order to specify a des- 
ired icon after the position designation cursor is 
matched with the desired icon displayed on the pic- 
ture screen of the CRT, a trigger operation for taking 
in the icon is executed by a central processing unit 
(CPU) of the computer. Further, when the user wants 
to enlarge the window of the desited icon, the mouse 
button is depressed one more time, or the mouse is 
moved to the position at which the window is enlarged 
on the tablet and the mouse button is depressed one 
more time at that position, the window of the icon 
thereby being enlarged as is commanded by the user. 

With such an information input control apparatus, 
the user must carry out two operations, namely the 
trigger operation for designating the desired icon and 
the operation for displaying the window of the icon at 
the predetermined position on the enlarged scale, 
which is very cumbersome for the user. Also, the 
mouse is too large for easily moving the cursor to the 
exact position of the desired icon properly. 

According to one aspect of the invention there is 
provided a data processing apparatus comprising: 

an input device and a display device mounted one 

upon the other; 

a pen; 

means for detecting whether or not a point of the 
pen is touched down; 

means for detecting whether or not data indica- 
tive of a coordinate position at which the point of 
the pen is touched down indicates that said posi- 
tion corresponds to a coordinate position at which 
a desired icon is displayed on the display device; 
means for controlling the icon display coordinate 
position such that the icon display coordinate 
position is moved in accordance with movement 
of the coordinate position of the touched-down 
point of the pen; 

means for detecting whether or not the coordinate 
position of the touched-down point of the pen is 
moved to a predetermined area; and 
means for converting the icon into a window when 



it is detected that the point of the pen is moved up 
within said predetermined area. 
According to another aspect of the invention there 
5 is provided a data processing apparatus comprising: 
an input device and a display device mounted one 
upon the other, 
a pen; 

means for detecting whether or not a point of the 

10 pen is touched down; 

means for detecting whether or not data indica- 
tive of a coordinate position at which the point of 
the pen is touched down indicates that said posi- 
tion corresponds to a coordinate position at which 

15 a desired window is displayed on the display 

device; 

means for controlling the window display coordh 
nate position such that the window display coor- 
dinate position is moved in accordance with 
20 movement of the coordinate position of the tou- 

ched-down point of the pen; 
means for detecting whether or not the coordinate 
position of the touched-down point of the pen is 
moved to a predetermined area; and 
25 means for converting the window into an icon 

when it is detected that the point of the pen is 
moved up within said predetermined area. 
A data processing apparatus embodying the 
inventions and described in detail hereinbelow can 
30 eliminate or reduce the afore noted shortcomings and 
disadvantages encountered with the prior art. Thus, 
the user can activate or deactivate a designated func- 
tion by dragging a pen. Also, a starting or ending posi- 
tion of a dragging operation can be designated 
35 naturally. Further, a position of a window, that is a 
position of, for example, an enlarged icon, can be 
determined with ease. 

The invention will now be further described, by 
way of illustrative and non-limiting example, with 
40 reference to the accompanying drawings, in which 
like reference numerals are used to identify the same 
or similar parts throughout, and in which: 

Figure 1 shows a data processing apparatus 
embodying the invention; 
45 Figure 2 is a block diagram showing hardware of 

the data processing apparatus; 
Figures 3A and 3B are views of a display screen 
of a display portion of the data processing 
apparatus; and 
so Figures 4A and 4B are flow charts illustrating the 
operation of the data processing apparatus. 
A data processing apparatus embodying the 
invention will now be described in detail with refer- 
ence to Figures 1 to 4. Figure 1 is a front (user) view 
55 of the data processing apparatus, in this case, the 
data processing apparatus is embodied, for example, 
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in the form of a so-called portable computer of very 
small type. 

As shown in Figure 1, the data processing 
apparatus has a lower portion including a display por- 
tion or apparatus 1 formed as a liquid crystal display 5 
(LCD) device or the like. The display portion 1 dis- 
plays thereon video image data from a central proces- 
sing unit (CPU) of the computer, which will be 
described later. An input apparatus, that is an input 
tablet 2 formed as a so-called transparent touch sen- 10 
sor or the like, is mounted on (e.g. stuck to) a surface 
of the display portion 1 . 

An electronic pen 3 is used to input desired data 
or the like in cooperation with the input tablet 2. The 
pen 3 incorporates therein a sensing or detecting 15 
means (not shown) for detecting a change of electro- 
static capacity between the input tablet 2 and the point 
of the pen 3. Coordinate position data of the pen 3 can 
be detected even when the point of the pen 3 is not 
brought into contact with the surface of the input tablet 20 
2, and a marker (not shown) is displayed on the dis- 
play portion 1 at a position thereon at which the point 
of the pen 3 is opposed to the input tablet 2. When the 
user holds the pen 3 and touches the input tablet 2 
with the point of the pen 3, that is, "touches down" the 25 
point of the pen, items of information such as a contact 
condition, the contact position and so on are supplied 
through a cord 4 or the like to the CPU, thereby input- 
ting desired data or the like. Signals can also be sup- 
plied to the CPU from predetermined key switches 5a, 30 
5b, 5c, 5d and 6a, 6b, 6c, 6d. 

A speaker 7 is provided on an upper portion of the 
data processing apparatus, together with a battery 
accommodating portion 8 which can accommodate a 
power source battery inside a lid which can be opened 35 
and dosed by a locking mechanism 9. A power switch 

10 powers the data processing apparatus when it is 
depressed. 

Dashed lines on the upper and lower portions of 
Figure 1 represent positions of mechanisms and the 40 
like provided at the side of the data processing 
apparatus. A dashed fine block 1 1 represents a micro- 
phone and dashed line blocks 12 represent operation 
keys that are used to effect solid state recording which 
will be described later. When a record operation key 45 
12 is operated, an audio signal fed to the microphone 

11 is recorded and emanates from the speaker 7. A 
dashed line block 13 represents a socket into which 
an integrated current (IC) card such as a read only 
memory (ROM), a random access memory (RAM) or so 
the Ike can be inserted and operatively connected to 

the apparatus. A dashed line block 14 represents the 
position of a socket which is used to connect the data 
processing apparatus with external apparatus when 
the user wants to expand the system of the data pro- 55 
cessing apparatus. 

A hinge 15 incorporates therein a flexible sub- 
strate or the like via which a signal can be transmitted. 



By means of the hinge 15, the data processing 
apparatus can be folded into its two (upper and lower) 
portions without being affected by difficulties from an 
electrical standpoint. When the data processing 
apparatus is in use, it measures about 29 cm long and 
about 20.5 cm wide in its unfolded state and is about 
4 cm thick in its folded state. The display portion 1 is 
about 15 cm wide and about 10 cm long. 

Figure 2 is a block diagram of the hardware incor- 
porated in the data processing apparatus. The CPU 
is designated 21 and, as outlined below, is connected 
to various function ICs by way of a bus line 22. 

A ROM 23 is connected to the CPU 21 and the 
ROM 23 stores therein, in addition to a system prog- 
ram for the entire system of the data processing 
apparatus, a Japanese syll a bary(kana)- Japanese 
character(kanji) conversion program for a word pro- 
cessor for the Japanese language and dictionary 
data, a recognition program for recognising hand-writ- 
ten input, and dictionary data and so on. 

A RAM 24 is used as a RAM work area and a 
backup RAM 25 is connected to a power source 26. 

A timer IC 27 is driven by the power source 26. 
The timer IC 27 counts a clock signal from a quartz 
oscillator 28 and always outputs data of date and time 
at each timing point. 

The hardware of the data processing apparatus 
will now be described in association with the connec- 
tion relations of the respective portions shown in Fig- 
ure 1. 

A display controller 29 is operative to control the 
display portion 1 . The display controller 29 is control- 
led by the CPU 21 and display data formed by the 
CPU 21 is written in a video RAM (V-RAM) 30. The 
data written in the V-RAM 30 is supplied to the display 
portion 1 through the display controller 29. 

A tablet interface circuit 31 is formed unitarily with 
the display portion 1 for the input tablet 2 and the pen 
3. Data written to the input tablet 2 or the like is sup- 
plied to the CPU 21 by means of the tablet interface 
circuit 31. 

A solid state recording processing circuit 32 is 
operative to process an audio signal from the micro- 
phone 1 1 in a predetermined manner, such as by sub- 
jecting it to analog-to-digital (A/D) conversion or the 
like, and the thus- processed signal is written in the 
backup RAM 25 via the CPU 21. Audio signal data 
stored in the backup RAM 25 is supplied to the pro- 
cessing circuit 32, in which it is processed in a pre- 
determined manner, such as by subjecting it to 
digital-to-analog (D/A) conversion or the like. The 
thus-processed signal is supplied to the speaker 7 via 
a mixer 33. 

A dialler circuit 34 is operative to generate an 
acoustic signal corresponding to a dual tone signal of 
a so-called push-phone when receiving data repre- 
senting a desired telephone number or the like from 
the CPU 21. This signal is supplied through the mixer 
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33 to the speaker 7. 

A key interface circuit 35 is supplied with a signal 
from the key switch group such as the above-des- 
cribed key switches 5a to 5d, 6a to 6d, 10 and 12 or 
the like either directly or by way of a power source cir- 5 
cuit 36. The signal, converted in the predetermined 
manner, is supplied from the key interface circuit 35 
to the CPU 21. Signals from the power switch 10, the 
key switches 6a to 6d and the operation keys 12 
(except a stop button) are supplied to the key interface 10 
circuit 35 through the power source circuit 36 so that, 
when these key switches are operated, the power 
source circuit 36 is actuated first and then information 
generated by key operation is supplied to the key 
interface circuit 35. 15 

An interface circuit 37 and an interface circuit 38 
are operative to couple the IC cards to the external 
apparatus upon expansion. The interface circuits 37 
and 38 are coupled to the sockets 1 3 and 14, respect- 
ively. 20 

In the above-described data processing 
apparatus, when the power switch 10 shown in Figure 

1 is depressed, groups 40 of icons which make vari- 
ous processing operations possible are displayed on 

the display portion 1 as shown in Figure 3A. When a 25 
desired processing operation, for example a dialler 
processing operation, is to be selected from the icon 
groups 40 for execution, the user touches an icon 41 
of one of the groups 40, on which a picture of a tele- 
phone is drawn, with the point of the pen 3. A cross- 30 
shaped position designating cursor 42 is displayed on 
the picture screen of the display portion 1 when the 
point of the pen 3 approaches the panel surface of the 
display portion 1 serving as the input tablet 2, whereby 
the user can visually confirm the exact position of the 35 
point of pen 3 on the input tablet 2 very clearly. Then, 
if the user moves (that is, drags) the point of the pen 
3 to a display position on the surface of the input tablet 

2 without breaking contact between the surface and 

the point of the pen after having touched the desired 40 
icon 41 with the point of the pen, and moves the point 
of the pen 3 away from the surface of the input tablet 

2, an icon (hereinafter referred to as a window), 
enlarged in the form of a processing display mode of 

the desired icon 41, is automatically displayed on the 45 
display portion 1 as shown in Figure 3B. 

This operation, which is carried out by the CPU 21 
shown in Figure 2, will now be explained more fully 
with reference to a flow chart shown as Figure 4A. 

Referring to Figure 4A, following the Start of oper- so 
ation, the CPU 21 determines in a decision step S1 
whether or not the user has touched down the point 
of the pen 3 on the surface of the input tablet 2. When 
the input tablet 2 is touched with the point of the pen 

3, a switch (not shown) provided within the pen 3 is 55 
operated, which provides an operation signal which is 
transmitted to the CPU 21 through the tablet interface 
circuit 31. If the input tablet 2 is not touched by the 



point of the pen 3, as represented by a NO output at 
the step S1, the routine returns to the step S1. If, on 
the other hand, the input tablet 2 is touched by the 
point of the pen 3, as represented by a YES output at 
the step S1, the routine proceeds to a decision step 

52. It is determined in the step S2 whether or not the 
point of the pen 3 has been set down within a pre- 
determined area occupied by the icon groups 40. If a 
NO is outputted at the step S2. the routine proceeds 
to a decision step S3. It is determined in the step S3 
whether or not the user is holding up the pen 3, that 
is whether the point has been moved up (lifted off) so 
that it is out of contact with the surface of the input tab- 
let 2. If a YES is outputted at the step S3, the routine 
returns to the step S1 . If a NO is outputted at the step 

53, the routine returns to the step S3. During this 
period, other operations, except for an operation in 
which the icon is enlarged (enlarged in the form of a 
window) are executed. If the point of the pen 3 is set 
or touched down within the area of the icon group 40 
as represented by a YES output at the step S2, the 
routine proceeds to a decision step S4 at which it is 
determined whether or not the pen coordinate posi- 
tion is shifted from a designated value (position) by a 
large amount. This decision is made by comparing the 
amount of shift of the pen coordinate position with a 
reference shift amount stored in memory, such as the 
ROM 23 or the like. Alternatively, it may be deter- 
mined in the step S4 whether or not the pen coordi- 
nate position is out of a designated area, as shown by 
an asterisk. In that case, the decision is made by com- 
paring the amount of shift with the designated area 
stored in the memory. 

If a NO is outputted at the step S4, the routine pro- 
ceeds to a decision step S5 at which it is determined 
whether or not the user is holding up the pen 3 so that 
its point is away from the surface of the input tablet 2. 
If a NO is outputted at the step S5, the routine returns 
to the step S4. If a YES is outputted at the step S5 t 
the routine returns to the step SI. If the pen coordinate 
position is considerably shifted from the reference 
amount or if the pen coordinate position is outside of 
the predetermined designated area (e.g. the area 
shown hatched in this embodiment) as represented 
by a YES output at the step S4, the routine proceeds 
to a step S6. In the step S6, the processing display 
form of the designated icon 41 is enlarged as a win- 
dow 43 shown in Figure 3B. Under this state the point 
of the pen 3 is touching the panel surface of the input 
tablet 2 so that, even if the user touches one of various 
processing menus displayed on the window 43, the 
CPU 21 is inhibited from being operated. 

Then, the processing of the CPU 21 proceeds to 
a decision step S7 at which it is determined whether 
or not the user is holding up the point of the pen 3 
away from the panel surface of the input tablet 2. If a 
NO is outputted at the step S7, the routine proceeds 
to a step S8. In the step S8, if the user wants to move 
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the window 43 as shown, for example, in Figure 3B, 
to a central position of the display portion 1, that is to 
move the window in the direction shown by an arrow 
B in Figure 3B, the user moves (drags) the point of the 
pen 3 in the direction of the arrow B whereby the large 5 
display icon, that is the window 43, is moved 
(dragged) to the position shown by a broken line in 
Figure 3B. Upon completion of the step, SB, the 
routine, of course, returns to the step S7. If it is deter- 
mined that the user is holding up the point of the pen to 
3 away from the pane! surface of the input tablet 2 as 
represented by a YES output at the step S7, the pro- 
cessing proceeds to a step S9 at which the icon is acti- 
vated so that the various processing menus within the 
window 43 can be executed. Then, the routine is 15 
ended. 

When the user wants to bring the large icon, that 
is the window 43 displayed on the display portion 1 as 
shown in Figure 3B, back to its original position, he 
touches a portion other than a function operation por- 20 
tion, for example a hatched upper edge portion 44 
within the window 43, with the point of the pen 3 and 
drags the point of the pen 3 back to the telephone icon 
41 of the original icon group 40 without breaking con- 
tact between the point of the pen and the tablet, 25 
whereby the icon of large size is dragged to be retur- 
ned to and stored in that position. To carry out this 
operation, the pen 3 need not always be returned to 
the telephone icon 41. Instead, the pen 3 can be retur- 
ned to a predetermined icon in the icon group 40, 30 
whereupon the window can automatically be stored in 
a vacant icon position. The above-mentioned oper- 
ation wKi be explained more fully with reference to the 
flow chart shown as Figure 4B. 

Referring to Figure 4B, following the Start of oper- 35 
ation, the CPU determines in a decision step ST1 
whether or not the user is touching the input tablet 2 
with the point of the pen 3. If a NO is outputted at the 
step ST1 , the routine returns to the decision step ST1 . 
If, on the other hand, a YES is outputted at the step 40 
ST1, the routine proceeds to a decision step ST2. It 
is determined in the decision step ST2 whether or not 
the point of the pen 3 is touching the large icon, that 
is the upper edge portion 44 of, for example, the win- 
dow 43 shown in Figure 3B and the point of the pen 45 
3 is located within the predetermined area except for 
the function button group of the window 43. If a NO is 
outputted at the step ST2, the routine proceeds to a 
step ST3 at which processing similar to that of the step 
S3 of Figure 4 A is performed. If the point of the pen 3 so 
is located within the window 43 as represented by a 
YES output at the step ST2. the routine proceeds to 
steps ST4 and ST5. The steps ST4 and ST5 are simi- 
lar to the steps S4 and S5 of Figure 4A. If a YES is 
outputted at the step ST4 or if the pen coordinate posi- 55 
tion is shifted from the designated value by a very 
large amount or if the point of the pen 3 is outside of 
the designated area, the routine proceeds to a step 



ST6 at which the icon (the window in this case) is 
reduced in size under the control of the CPU 21. 

Next, in a decision step ST7, it is determined 
whether or not the user is holding the pen 3 up away 
from the panel surface of the input tablet 2. If a NO is 
outputted at the step ST7, the routine proceeds to a 
step ST8 at which the icon (displayed as a reduced 
size icon) is moved, and the routine returns to the step 
ST7. If it is determined in the step ST7 that the user 
is holding the pen 3 up away from the panel surface 
of the input tablet 2, the routine proceeds to a step 
ST9 at which the icon is deactivated, after which the 
routine proceeds to a step ST 10 in which the icon is 
moved to the predetermined vacant position and the 
processing is ended. 

Since the data processing apparatus is construc- 
ted and operated as described above, the user can 
activate or deactivate a designated function merely by 
dragging the pen 3. Further, the starting and ending 
positions of the dragging of the pen 3 can be desig- 
nated naturally, and the window position or the icon 
position to be enlarged can be determined with ease. 

As set out above, the icon and the window can be 
designated by one operation in which the user 
touches the panel surface of the input ta btet 2 with the 
point of the pen 3 and drags it along the panel surface 
of the input tablet 2 so that cumbersome operation is 
avoided, and the starting and ending of the dragging 
operation of the pen 3 can be carried out very clearly. 
Furthermore, various other effects, such as easy desi- 
gnation of a window and so on, can easily be 
achieved. 



Claims 

1. A data processing apparatus comprising: 

an input device (2) and a display device (1) 
mounted one upon the other; 
a pen (3); 

means (21-S1) for detecting whether or not a 
point of the pen (3) is touched down; 
means (21 -S2) for detecting whether or not 
data indicative of a coordinate position at 
which the point of the pen (3) is touched down 
indicates that said position corresponds to a 
coordinate position at which a desired icon 
(e.g. 41 ) is displayed on the display device (1); 
means (21) for controlling the icon display 
coordinate position such that the icon display 
coordinate position is moved in accordance 
with movement of the coordinate position of 
the touched-down point of the pen (3); 
means (21 -S4) for detecting whether or not 
the coordinate position of the touched-down 
point of the pen (3) is moved to a predeter- 
mined area; and 

means (21-S6/S7/S9) for converting the icon 



BNSDOCIO: <EP 0457990 A2_l_> 



9 EP 0 457 990 A2 10 

(41) into a window (43) when it is detected that 
the point of the pen (3) is moved up within said 
predetermined area. 



2. A data processing apparatus comprising: 5 
an input device (2) and a display device (1 ) 
mounted one upon the other; 
a pen (3); 

means (21-ST1) for detecting whether or not 
a point of the pen (3) is touched down; 10 
means (21-ST2) for detecting whether or not 
data indicative of a coordinate position at 
which the point of the pen (3) is touched down 
indicates that said position corresponds to a 
coordinate position at which a desired window 15 
(e.g. 43) is displayed on the display device (1 ); 
means (21) for controlling the window display 
coordinate position such that the window dis- 
play coordinate position is moved in accord- 
ance with movement of the coordinate 20 
position of the touched-down point of the pen 
(3); 

means (21-ST4) for detecting whether or not 
the coordinate position of the touched-down 
point of the pen (3) is moved to a predeter- 25 
mined area; and 

means (21-ST6/ST7/ST9) for converting the 
window (43) into an icon (e.g. 41) when it is 
detected that the point of the pen (3) is moved 
up within said predetermined area. 30 
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@ Information is inputted to a data processing 
apparatus by using a pen (3) in conjunction with 
an input device or tablet (2) which is mounted 
on a display device or panel (1). If the user 
touches an icon (e.g. 41), selected from one or 
more groups (40) of icons displayed on the 
display device (1) and indicating various func- 
tions, with the point of the pen (3), drags the 
pen (3) along the surface of the display device 
(1) to a display position, and then releases the 
point of the pen from the surface of the display 
device, an enlarged processing display form of 
the icon (41), in the form of a window (43), is 
automatically displayed at a desired position on 
the display device (1). Further, if the user 
touches the window (43) with the point of the 
pen (3) and drags the pen along the surface of 
the display portion to the position of a desired 
icon, the enlarged window is automatically re- 
duced and is stored in a vacant icon. 
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